%  Copyright (C) 2002 Regents of the University of Michigan, portions used with permission 
%  For more information, see http://csem.engin.umich.edu/tools/swmf
\section{Installation and Compilation \label{section:install_compile}}

\subsection{Directory Structure \label{section:directory}}

Upon installing \BATSRUS, you should find the following files
and directories inside the main \BATSRUS\ directory.  
The subdirectories are included without listing
all of the files that they contain.
\begin{verbatim}
File or Directory:         Notes:

Config.pl                  Script to (un)install and configure the code
Configure.pl               Script to configure the source code
Configure.options          Default configuration options
Doc/                       Documentation 
   Tex/                       Latex source code for documentation
   HTML/                      HTML form of the documentation
Idl/                       Source code for IDL post processing
Makefile                   Main makefile
Makefile.def               Place holder for the configured Makefile.def
PARAM.XML                  Description of input parameters in text and XML
PARAM.pl                   PARAM.XML converted to a Perl file
Param/                     Sample PARAM.in files, include files
Scripts/                   Contains scripts to run the code on
   ConvertRestart.pl          Script to convert the endianness of restart files
   IDL/                       Scripts used to process IDL files
   Run/                       Scripts used to run BATSRUS and process output
      AIX/                       -IBM SP
      Darwin/                    -Macintosh
      IRIX/                      -SGI Origin
      IRIX64/                    -SGI Origin
      Linux/                     -Linux (Beowulf clusters, SGI Altix)
      OSF1/                      -Compaq
   TEC/                       Scripts used to process TECPLOT files
share/                     Scripts and source code shared with SWMF
src/                       Source code for BATSRUS
srcInterface/              Source code for interfaces with other components
srcPostProc/               Post processing source code
TestBatsrus.pl             Script to test BATSRUS
TestCompare.pl             Script to compare test results
TestCovariant              Script to test BATSRUS in 'covariant' configuration
TestParam.pl               Script to test the input parameter file
TestSuite.pl               Script to run a whole test suite
util/                      Utilities NOMPI and TIMING shared with SWMF
\end{verbatim}

\subsection{Setting Grid Structure Before Compiling \BATSRUS\ \label{section:before_compile}}

Part of the design of \BATSRUS\ is the conscious decision to limit the
usage of allocateable variables.  This is because
allocateable memory was found to slow the code considerably and 
the choice was made to maximize code speed.  This means that the
most of the arrays of \BATSRUS\ must be dimensioned with a fixed size
in the source code. 
Since platforms vary widely, there is no way to set these in a universal
way and the user will have to set them.  There are two main sets of 
parameters that must be changed.

\subsubsection{Number of Blocks per Processor: {\tt nBLK} \label{section:nblk}}

As described in the DESIGN document, the main unit of mesh used for
computation in \BATSRUS\ is the Cartesian block.  The major part of
memory is dimensioned in this block structure.  Therefore, the number
of blocks basically determines the amount of memory the code will use.
The number of blocks that any given processor can handle depends on
the amount of memory that each node has available and must be changed
in the file {\tt src/ModSize.f90}. This is normally done with the {\tt
Config.pl} script.  The maximum number of blocks is defined by the
variable {\tt nBLK}.

\subsubsection{Number of Cells in Each Block: {\tt nI},
{\tt nJ}, {\tt nK} \label{section:ncells}}

Typically, \BATSRUS\ is run with blocks that are cubes, 
but this does not have to be the case.  
The size of blocks is determined in {\tt ModSize.f90} by the parameters 
{\tt nI}, {\tt nJ}, {\tt nK}, the numbers of cells in a block
in each dimension.  This number does not include ghost cells which the code
takes care of automatically.  The user should set these numbers with the 
Config.pl script according to their needs based on the following restrictions.
\begin{itemize}
\item {\tt nI,nJ,nK} variables should not be less than 4 and must be set as even
      integers (4, 6, 8, 10 ...). If the grid is uniform (all cells the same size)
      the code should run with {\tt nI,nJ,nK} set to 2.
\item Smaller blocks means a larger ratio of ghost cells to computational cells.
      For example a 4x4x4 block has a 64 cell computation region, while
      it has 8x8x8 = 512 total cells counting ghost cells.  In other words,
      the majority of the storage is ghost cells (\#Ghost/\#Computation = 7).
      If the user used 8x8x8 blocks, the computation region would have 512 
      cells while the total number of cells is 1728 and the ratio of ghost cells
      to computation cells is only 2.4.
\item The time the code spends message passing depends on the number of
      ghost cells in a block and on the number of blocks.  With larger blocks
      the message passing time may be reduced.
\item Larger blocks mean more wasted cells when doing AMR to resolve
      features of the solution.  If the user uses 4x4x4 block he or she will
      have much better control over where the resolution is located.  With larger
      blocks the user will have to resolve a larger area to get the interesting
      area resolved.
\end{itemize}
The user should note that as usual, in numerical simulations there is a trade off
in efficiency in resolving the solution and the amount of storage.  In general,
larger blocks mean less ``wasted'' storage but will inevitably lead to more cells
to resolve the same features.

\subsubsection{Getting the Grid you Want \label{section:proc_dims}}

The initial grid (before any refinement) contains
a number of blocks determined by the {\tt proc\_dims(i)} variable, where
{\tt i} is the direction (1=x, 2=y, 3=z).  If the values are set to
\begin{verbatim}
  proc_dims(1) = 2
  proc_dims(2) = 1
  proc_dims(3) = 1
\end{verbatim}
then the top level blocks form a brick consisting of 2 blocks arranged
along the x axis. This variable is called {\tt proc\_dims} because in
the original design of \BATSRUS\ the initial number of blocks could
not exceed the number of processors.  In the current version there is
no such restriction, and the name is kept for historical reasons
only. A more descriptive name would be {\tt nRootBlock\_D} which
refers to the number of root blocks in the 3 coordinate directions.

The sizes of cells in these top level blocks is determined by the
initial number of blocks and also by the physical size of the
computational domain and the values of {\tt nI}, {\tt nJ}, {\tt nK}.
As an example, we show a section of the input parameters used when
running the code
\begin{verbatim}
#GRID        
4                      nRootBlockX
2                      nRootBlockY
1                      nRootBlockZ
-32.                   xMin
 32.                   xMax
-8.                    yMin
 8.                    yMax
 0.                    zMin
 4.                    zMax
\end{verbatim}
These input parameters, along with  
{\tt nI,nJ,nK} define the initial grid used in the simulation.
The values xMin, xMax, ... indicate the physical 
domain of the computation.  In section~\ref{section:ncells}
we indicated that the number of cells in a block in each direction is determined
in {\tt ModSize.f90}.  Here we determine the actual shape of the computational
domain as well as the actual shape of each cell. Typically, \BATSRUS\ is
run with cells and blocks that are cubes.  While this is not required, 
cells that have large aspect ratios may lead to less accurate computations.  
The shape of a cell is
determined by the {\tt nI,nJ,nK}, the {\tt proc\_dims} and the physical size of 
each dimension. The size of a cell in the top level blocks is given by 
\begin{equation}
  dx = \frac{ \mbox{xMax}-\mbox{xMin} }{\mbox{nRootBlockX}*\mbox{nI}} \qquad
  dy = \frac{ \mbox{yMax}-\mbox{yMin} }{\mbox{nRootBlockY}*\mbox{nJ}} \qquad
  dz = \frac{ \mbox{zMax}-\mbox{zMin} }{\mbox{nRootBlockZ}*\mbox{nK}}
\end{equation}
We give three examples.
First, with 4x4x4 blocks, if the physical dimension
of the computation region is a cube, then setting the initial number of blocks
in each dimension ({\tt proc\_dims}) equal will ensure cells that are cubes.  
For the second example, if one dimension is twice
as large as the other two, then beginning with twice as many blocks in this dimension
will again ensure cubic cells.  
Finally, combining xMin, xMax, yMin, yMax, zMin, zMax, nI, nJ, nK and proc\_dims 
allows the user to make computation
regions that are highly stretched in one dimension while still having
cubic cells.  If the user wanted to do a two dimensional problem, for example,
he or she could choose parameters to minimize the numbers of cells in 1 dimension.
Figure~\ref{fig:stretched_initial_grid} shows a 2 dimensional view of a 
grid defined using the input shown above and 
with {\tt nI} = 16, {\tt nJ} = 8, {\tt  nK = 4}.
In the figure, the dark lines show the locations of blocks, initially
4x2x1 (the z dimension is not shown).  This grid would have to be 
run on at least 8 processors.  The
The lighter lines show the individual cells in one of the blocks.  These are 16x8x4 
(again, the z dimension is not shown).  Notice that the cells are cubes (dx=dy=dz=1.0),
but the computation region is stretched in the x and y directions.
\begin{figure}
\begin{center}
\includegraphics*[width=6.0in]{proc_dims.pdf}
\end{center}
\caption{Initial block and grid structure for {\tt proc\_dims} of 4x2x1
and {\tt nI,nJ,nK} of 16x8x4.  The z dimension is not shown.  Heavy line indicate
blocks, lighter lines indicate individual cells.
\label{fig:stretched_initial_grid}
}
\end{figure}

{\bf THERE IS ONE OTHER RESTRICTION ON THE WAY THAT GRIDS CAN BE CREATED
WHEN USING A CENTRAL BODY.}
Because the first body is created with its center at the origin, it is
important that eight block corners meet at the origin.  The user must 
think ahead so that the final desired refinement level at the body
satisfies this restriction.  While
the code may run if this is not that case, Tecplot output will not be
correct, initial refinement may not do what you thought it should and
there may be other problems.  
We give parameters for the standard Earth case as an example
\begin{verbatim}
#GRID        
2                      nRootBlockX
1                      nRootBlockY
1                      nRootBlockZ
-224.                  xMin
 32.                   xMax
-64.                   yMin
 64.                   yMax 
 64.                   zMin
 64.                   zMax
\end{verbatim}
Initially there are two blocks which meet at x=-96. 
The origin does not lie at block corners.
Two refinements of these initial blocks will create 128 blocks and
will give the required corners at the origin.  Since this case is
never run with a lower resolution than this we are okay.


\subsection{The Main Makefile \label{section:main_make}}

The main executable and all the post processing executables along
with run directories can all be set up from the main makefile.
This is true for the framework and the stand alone code as well,
and the makefile targets are very similar in both cases.
Typing
\begin{verbatim}
  make help
\end{verbatim}
lists the available executables, directories and scripts which can be built 
or executed. The rest of this subsection is specifically about the stand 
alone code, i.e. the use of the Makefile in the \BATSRUS\ main directory.
In stand alone mode the help list is the following: 
\begin{verbatim}

  You can ``make'' the following:
  
    <default> BATSRUS in stand alone mode, help in SWMF
  
    install   (create MAKEFILES, src/ModSize.f90, src/mpif90.h)
    MAKEFILE_DEF (create/correct Makefile.def)
  
    LIB     (Component library libGM for SWMF)
    BATSRUS (Block Adaptive Tree Solar-Wind Roe Upwind Scheme)
    NOMPI   (NOMPI library for compilation without MPI)
    PIDL    (PostIDL program creates 1 .out file from local .idl files)
    PSPH    (PostSPH program creates spherical tec file from sph*.tec files)
  
    help      (makefile option list)
    clean     (rm -f *~ *.o *.kmo *.mod *.T *.lst core)
    distclean (make clean; rm -f *exe Makefile Makefile.DEPEND)
    dist      (create source distribution tar file)
  
    PDF       (Make PDF version of the documentation)
    HTML      (Make HTML version of the documentation)
  
    rundir      (create run directory for standalone or SWMF)
  
    mpirun      (make BATSRUS and mpirun BATSRUS.exe on 8 PEs)
    mpirun NP=7 (make BATSRUS and mpirun BATSRUS.exe on 7 PEs)
    mprun NP=5  (make BATSRUS and mprun  BATSRUS.exe on 5 PEs)
  
    spherical_src  (Make SPHERICAL directory with BATSRUS on spherical grid)
    spherical_conf (Make SPHERICAL directory and link it to BATSRUS_conf)
    covariant_src  (Make COVARIANT directory with BATSRUS on covariant grid)
    corelax_src    (Make CORELAX directory for the covariant version
                of the magnetogram-driven solar wind)
    cartesian_src  (removes source code for covariant grid)
    relax_src      (Make RELAX directory for the Cartesian
                    version of the magnetogram-driven solar wind)

\end{verbatim}
As an example, in order to make {\tt BATSRUS.exe}, simply type 
\begin{verbatim}
  make
\end{verbatim}
To make a distribution (tar file) of the code like the one used for installation,  type
\begin{verbatim}
  make dist
\end{verbatim}
To make a directory with everything setup to run code
\begin{verbatim}
  make rundir
\end{verbatim}
The executables reside in the {\tt src} directory.  When a run
directory is created it is located in the root directory of the
installation and will have links to the executables, will have copies of
the appropriate scripts for the current platform and will have the
directory structure for running the code setup.

The Makefile in the root directory of the distribution calls the
different makefiles in the  {\tt src/}, {\tt srcPostProc/}, {\tt share/}
and {\tt util/} directories in order to do the actual compilations. 

\subsection{Compiler Flags \label{section:compiler_flags}} 

The makefile {\tt Makefile.conf} in the main directory (of SWMF or
the stand alone \BATSRUS) contains the system dependent flags.
In this file, the user will find several sets of commented out flags.
The default version will be the one that should be used for running 
the code to do production type runs.  Other sets of flags include
those for debugging (which does more error checking), among others.
As mentioned above, compile flags are not trivial to set and if the user 
has to change them, they will
likely have to consult the manual pages of the F90 compiler.

One main flag, {\bf PRECISION}
determins the number of bytes a real number occupies.  
On most systems the default size real is 4 bytes.  
In the Makefiles, the flag {\bf PRECISION} can be changed to
indicate the number of bytes to use for a real.
When debugging it is often necessary to use 8 byte real to differentiate
round off error from true errors.  For production runs, 8 byte reals should
be more accurate, but may not always be necessary. 
Using 8 byte real will double the size of the code.
The trade off is of course loss of some accuracy with 4 byte reals
and increased code and restart file size for 8 byte reals.
In addition, on 32 bit machines (such as most PC's and Suns), the code
will run slower (about 30 \% on our PC based Beowulf) when using 8 byte reals.

Another compiler flag that the user may need to change is the {\bf MPILIB}
flag.  As mentioned in section~\ref{section:compiler}, the code can be
run on a single processor with our {\tt NOMPI} library, which the user
can make (see section~\ref{section:main_make}).  To run with this library the
{\bf MPILIB} needs to be changed from calling the system libraries to 
the NOMPI library.  As an example, the {\tt Makefile.IRIX64}, is the makefile
for the SGI Origin.  In this makefile the user will find the following lines:
\begin{verbatim}
MPILIB = -lmpi
#MPILIB = -L. -lNOMPI
\end{verbatim}
To use the {\tt NOMPI} library, the first line must be commented out (place a 
\# in front of the line) and the \# should be removed from the second line.
With the {\tt NOMPI} library the code can run on a single processor only.

\subsection{Run Directory Structure \label{section:run_dir}}

The {\tt run} directory is setup with {\tt make rundir}.
The purpose for creating run directories is to separate runs.  
You can rename them
\begin{verbatim}
  mv run run_CME
\end{verbatim}
to do a simulation of a CME, for example.  
You can also move the {\tt run} directory to another
directory or disk, but this is not recommended,
because it will be difficult to relate the source code
and the actual run.

When /BATSRUS/ is used inside the framework, it gets a
subdirectory named by the component. For example
if /BATSRUS/ is used as the SC component, it 
will read files from and write files into the
\begin{verbatim}
run/SC
\end{verbatim}
directory. The following directories links and files 
are created in {\tt run/SC}:
\begin{verbatim}
IO2/
restartIN/
restartOUT/
PostIDL.exe -> /home/USER/SWMFDIR/bin/PostIDL.exe
PostSPH.exe -> /home/USER/SWMFDIR/bin/PostSPH.exe
pIDL
pTEC
\end{verbatim}
In the stand alone mode of \BATSRUS\ the subdirectory name
is always GM (independent of the value of the STANDALONE variable
in Makefile.def). To facilitate access to the subdirectories and
scripts, a symbolic link is provided
in the {\tt run} directory to all the items in {\tt run/GM}.

In SWMF the run directory itself contains all the component 
subdirectories, and the following files, links and subdirectories
\begin{verbatim}
core
job.MACHINE
LAYOUT.in
Param       -> ../Param
PARAM.in
SWMF.exe    -> /home/USER/SWMFDIR/bin/SWMF.exe
STDOUT/
\end{verbatim}
In the stand alone \BATSRUS, other than the links to the
items in the run/GM subdirectory, the following files and links
are created
\begin{verbatim}
BATSRUS.exe -> /home/USER/bats/src/BATSRUS.exe
core
job.MACHINE
PARAM.in
\end{verbatim} 
The 'core' file is there to prohibit the creation a huge
core file due to a run time error. The job.MACHINE (if any)
file contains an example script to submit a job on the
given MACHINE. The PARAM.in file
is copied from Param/PARAM.DEFAULT and should be
normally modified or replaced before running the code.

This directory is now setup to run the code.  Links have been made to 
{\tt BATSRUS.exe} which is the main MHD executable, as well as the 
IDL post processing executable {\tt PostIDL.exe} and the Tecplot post 
processing executable {\tt ProcessIO.exe}.  The script {\tt pTEC}
is for processing the Tecplot files while {\tt pIDL} is for
processing the IDL plot files.  The scripts {\tt job.XXXXX} contain
sample scripts for submitting jobs to the various machines that we
have run the code on.  

The {\tt run} directory contains subdirectories or links to directories
such as {\tt restartOUT/}, {\tt IO2/}, and {\tt Param/}.  
The first three are needed to run the code, 
while the fourth contains input files which can be
included from the {\tt PARAM.in} file, as described below.
Normally one needs at least the {\tt IO2/} and {\tt restartOUT}
directories to store plot files and restart files.
These are the default names, which can be changed in the PARAM.in
together with the directory names.
If a needed subdirectory is missing, the code will stop with an error
message.

{\tt IO2/} is the most important directory, since it contains output
of the MHD run.  The {\tt restartOUT/} directory is where restart
files are written.  These files allow the code to be restarted if it
either crashes or if the code must be run on a queue system in which
the run takes longer than the length of the run time allowed on the
machine.  For example, on some machines the codes are allowed to run for
two hours.  Since most time accurate runs take much longer than
this, restart files can be saved near the end of the 2 hours
and the code can be restarted after waiting in the queue again.

The {\tt restartIN/} directory is needed to actually restart the run
from a previous run.  Often the {\tt restartIN/}
directory is simply a link to the {\tt restartOUT/} directory.  This, however,
can be dangerous because if something ``bad'' occurs during the
writing of the restart files, the last save may be destroyed.  It is
wiser to move the {\tt restartOUT/} directory to some other
location, link the {\tt restartIN/} directory to this directory, and
create a new {\tt restartOUT/} directory.  This will ensure that the
old save will be secure, while restarting correctly.

The {\tt PARAM.in} file contains the input parameters for the \BATSRUS\ 
code. The details about this file and the input parameters
are given in section~\ref{chapter:param}.
